Add Book to My BookshelfPurchase This Book Online

Chapter 8 - Troubleshooting

Cisco TCP/IP Routing Professional Reference
Chris Lewis
  Copyright © 1999 The McGraw-Hill Companies, Inc.

Troubleshooting the First Three ISO Layers
In this section, we'll review the process of troubleshooting internetwork problems that was first introduced at the end of Chap. 3. This will not prepare us to do much real troubleshooting work, but does get the thought process started off in the right direction. The real troubleshooting work starts when we consider the individual cases that follow.
The hypothetical problem we have to solve in the review taken in this section is that router 1 cannot ping router 3 via router 2 on the simple internetwork shown in Fig. 8-1.
Figure 8-1: Three router setups for basic troubleshooting review
We will consider simple troubleshooting techniques at each of the first three layers of the ISO data communications model. In practice, you would troubleshoot a connectivity problem such as this one link at a time from source to destination, testing the Physical, Data Link, and Network layer operations in turn, attempting to identify the point where the communication is failing. We consider each layer separately here as an introduction to troubleshooting techniques and to highlight that similar commands display different information for different interface types.
Simple Troubleshooting at the Physical Layer
Troubleshooting the Physical layer deals with issues such as:
  1. Are the devices physically connected in an appropriate way?
  2. Are interconnecting cables correct?
  3. Are the CSU/DSU devices configured and operating properly?
So the problem we have to solve is that router 1 cannot ping router 3. Having stated the problem, we need to gather information, which we will do beginning with the Physical layer issues. The first thing we can do at the Physical level is check to determine whether the devices are physically connected. This is okay if the routers are all in one location, since we can just take a look, but if they are geographically dispersed, we need to use displays generated by the Cisco IOS to tell us the state of physical connections. Imagine router 1 and router 2 are local in New York, and router 3 is in Atlanta.
Seeing whether the Ethernet interfaces are connected for router 1 and router 2 should be straightforward. Cisco routers typically have an AUI connector for an Ethernet interface, and as most cabling these days is based around twisted-pair cables, a transceiver is used to convert media from the AUI 10Base5 to 10Base-T. A transceiver has an LED display to tell you if it is connected to the router interface and whether it is receiving traffic from a hub.
So let's think about what could happen on a physical connection level to stop things from working, and then see how Cisco IOS can or cannot report the condition, which is necessary if we want to troubleshoot a remote connection such as the Ethernet 0 interface on router 3.
  1. Transceiver disconnected from Ethernet interface.
  2. Transceiver connected to Ethernet interface, but disconnected from hub.
  3. Ethernet interface connected to hub on which it is the only device.
We generally can gather pertinent information by using one or more of the following: a show command, a debug command, or a ping/trace command. We know that our problem is that router 3 is not reachable from router 1 by using the ping command, so let's start the troubleshooting process with the show command on the Ethernet interface of router 1, as illustrated in Fig. 8-2.
router1 #show int eO
Ethernet0 is up, line protocol is up
Hardware is Lance, address is 0000.0c47.42dd (bia 0000.0c47.42dd)
Internet address is 164.7.1.67 255.255.255.224
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/255, load 1/255
Encapsulation ARPA, loopback not set, keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 4:00:00
Last input 0:00:01, output 0:00:00, output hang never
Last clearing of "show interface" counters never
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 8000 bits/sec, 4 packets/sec
5 minute output rate 8000 bits/sec, 4 packets/sec
  717 packets input, 297281 bytes, 0 no buffer
  Received 567 broadcasts, 0 runts, 0 giants
  0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
  0 input packets with dribble condition detected
  3756 packets output, 475281 bytes, 0 underruns
  0 output errors, 0 collisions, 1 interface resets, 0 restarts
  0 output buffer failures, 0 output buffers swapped out
Figure 8-2: Output of the show interface ethernet 0 command
The first place to look on this display is at the status of the interface and the line protocol. Generically, a show command will display the interface as up, down, or administratively down; the line protocol will either be up or down. "Administratively down" means that the shutdown command has been entered into the configuration of the interface and needs to be removed with the no shutdown command. With LAN interfaces, the interface will never show a down condition unless something is seriously wrong with the router hardware. We will consider the case in which an asynchronous interface can report a down condition due to a misconfigured interface in the section on troubleshooting asynchronous communications.
For LANinterfaces, we are left with using the status of the line protocol to tell us if the physical connections are made. With the transceiver disconnected from the AUI interface, the line protocol is down. With the transceiver connected to the AUI interface, but disconnected from the hub, the line protocol report is still down. With the router the only device connected to the hub, the line protocol reports up.
At this stage, the only other command to look at is the media-type command if you are using a 4000-series router. This command can configure an interface for either 10Base-T or AUI operation; this command, set in interface configuration mode, must match the type of connection physically made to the router interface.
As you can see, we have limited remote monitoring capabilities to tell whether the physical connections are made for LAN connections. The best we can do remotely is see if the line protocol is up or down. If it is down, we have to get someone local to check the physical connections, because the only way to bring it up is to connect the interface to a working hub.
Before we move on to checking the status of the physical connections for the serial interfaces, we should try to determine if the router is connected to the correct hub to reach the devices it must reach. We do this by pinging the IP addresses of the other devices that we expect to be on the same segment. If we are operating in a Cisco IPX network environment, we can also issue IPX ping commands. Not all Novell servers support this functionality, so it cannot be used as a general troubleshooting tool such as the IP ping utility.
Now let's look at using Cisco IOS to check physical connectivity of the serial interfaces in use. In practice, I would now move on to checking the Data Link and Network layer issues with the Ethernet interfaces before moving on to the serial interfaces. For this introduction, however, it is useful to contrast the function of the show interface command for Ethernet and serial interfaces.
The show interface serial 0 command issued on router 2 produces the display shown in Fig. 8-3.
router2#show int serial 0
Serial0 is up, line protocol is up
Hardware is HD64570
Internet address is 164.7.1.97 255.255.255.224
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec, rely 255/255, load 1/255
Encapsulation HDLC, loopback not set, keepalive set (10 sec)
Last input 0:00:01, output 0:00:00, output hang never
Last clearing of "show interface" counters never
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 4000 bits/sec, 3 packets/sec
5 minute output rate 4000 bds/sec, 3 packets/sec
336 packets input, 226009 bytes, 0 no buffer
Received 95 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 ftame, 0 overrun, 0 ignored, 0 abort
3372 packets output, 293703 bytes, 0 underruns
0 output errors, 0 collisions, 1007 interface resets, 0 restarts
0 output buffer failures, 0 output buffers swapped out
12 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
Figure 8-3: Output of the show interface serial 0 command on router 2
The main points of interest regarding the physical connections are the status of the interface, the line protocol, and the status of the EIA signals given at the bottom of the display. Let's first consider what might be wrong at the Physical level and consider how we might use this display to detect these problems. At the Physical level, we will want to detect the following:
  1. Is the serial interface connected to a working CSU/DSU device?
  2. Is the local CSU/DSU communicating properly with the remote CSU/DSU?
  3. Is the remote CSU/DSU connected to the remote router serial port?
Again we want to see the show interface serial 0 command report that the interface is up and the line protocol is up. First, what will happen if the serial interface on router 3 is disconnected from the local CSU/DSU? The interface reports down and the line protocol also reports a down state. We also see that the EIA signals DCD, DSR, and CTS disappear. EIA signals are either generated from the router interface or the CSU/DSU device. In this case, the CSU/DSU generates DCD (carrier detect), DSR (data set ready), and CTS (clear to send). If these signals are not present, we know that the cable between the interface and the CSU/DSU is disconnected, or that the CSU/DSU is powered off.
Assuming that the correct cable is in use and that the CSU/DSU is powered on and connected to the router serial interface, how do we tell if the connection between the local and remote CSU/DSU is good? In a case in which the line between local and remote CSU/DSU has failed, we look at the show interface serial 0 output again, and see that all the EIA signals are present except the DCD signal. This tells us that the router-to-local-CSU/DSU connection is made, but there is no carrier signal between the CSU/DSU devices. This is the way that a router typically reports that a leased-line connection has gone down.
If we can get to the stage at which the show interface serial 0 command shows all signals present, the last thing to check on the Physical level is the remote CSU/DSU-to-remote-router interface connection. If you have a way of logging into the remote router 3, such as a dial-up connection to the console port, you can issue the show interface serial 0 command to see if the EIA signals are present, thus confirming the remote CSU/DSU-to-router interface connectivity. Without a way to log into router 3, you have to rely on local personnel to confirm connections and that devices are powered up.
As you can see, at the Physical layer we can get useful information from Cisco IOS for serial connections, but limited information for Ethernet connections.
Simple Data Link Layer Troubleshooting
Again, we will look at the Data Link layer issues for both Ethernet and serial interfaces to contrast the information that similar commands present for different interface types. At the Data Link layer, the protocols are concerned with forming frames and synchronizing devices prior to the exchange of data.
For Ethernet interfaces, we are mainly concerned about confirming that the interface has the same encapsulation as the other devices with which it must communicate. We can see the encapsulation used either by examining the router configuration, or by looking at the show interface ethernet 0 command and reading the fifth line of the output. There are four options for Ethernet encapsulation; other networking systems, unfortunately, know them by different names. A comparison of Cisco and Novell terminology is given as follows:
  ARPA is known to Novell systems as the ethernet_ii frame type. This is a common frame type for implementing IP on NetWare-based networks.
  SNAP is known to Novell systems as frame type ethernet_snap. The SNAP encapsulation is more common in Token-Ring environments.
  SAP is known as ethernet_802.2 and is the default frame type for NetWare 3.12, 4.x, and later servers.
  Cisco's term for Novell's 802.3 is novell-ether. Novell implemented its own frame type, 802.3, when NetWare was designed. This caused some confusion in the world of support personnel. IEEE 802.3 can support many layer 3 protocols; Novell 802.3, however, supports only IPX at the Network layer. Cisco avoids this confusion by referring to Novell's 802.3 as novell-ether.
Before you can proceed any further, the encapsulation on the interfaces must match that of the other devices with which it is to communicate. For Ethernet devices, that is all we need to check for this problem. In the section on troubleshooting Ethernet, we will look at other Data Link layer issues such as undersized or oversized packets and collisions.
For serial interfaces, the encapsulation must be the same on the communicating interfaces. The best way to check this is by looking at the configuration of each connected interface. Encapsulation on serial interfaces can be one of the following:
  ATM-DXI
  Frame relay
  HDLC
  LAPB
  PPP
  SMDS
  X.25
For serial interfaces, we can check to make sure the clock signal is operating correctly by issuing the show controllers serial 0 command. This display will show the speed of the clock signal received from the CSU/DSU device, if present. This is important, because if the clock signal is not there, the serial interface and CSU/DSU cannot communicate.
The clock signal is one area in which Ethernet and serial interfaces differ considerably. With serial interfaces, a constant clock signal synchronizes router interface to CSU/DSU communications. On an Ethernet interface, there is no such synchronization signal. Each packet sent on Ethernet uses the preamble at the front of the Ethernet frame to synchronize all devices on the Ethernet network as the frame is transmitted.
Simple Troubleshooting at the Network Layer
An Ethernet interface will show the line protocol as up, even if there are no other devices on its network segment; it does not need to see another IP device. It's different with serial interfaces. Before the line protocol comes up, serial interfaces need to see a device on the other end of the serial link that has an IP address in the same network number or subnet for which its serial interface is configured.
This is an important point. In order for the network or subnet number for which a serial interface is configured to appear in the router's routing table, the serial interface's line protocol must be up. So even if everything is correct at the Physical and Data Link layers, the routing table will not have the necessary entries to allow router 1 to ping router 3 if the router 1 and router 2 Serial 0 interfaces are not addressed for the same subnet.
Network layer troubleshooting is based around viewing the routing table on the routers along the path from source to destination, along with examining the IP addresses (or other protocol IDs) of those router interfaces. Typically, if routes do not appear in a routing table, you will check to see that appropriate static, default, or dynamic routes have been entered and then either configure any missing routes manually, or troubleshoot any dynamic routing process, such as RIP or IGRP, to get the routing tables updated.
As we have just shown, if a router's line protocol is down, all the routes that were accessible through that interface will be eliminated from the routing table. This is a good justification for following the process of Physical and Data Link troubleshooting before beginning Network layer troubleshooting. If you jump straight into Network layer troubleshooting, you could start to investigate a routing protocol issue, whereas the real problem might be, for example, with a line protocol down due to a missing CSU/DSU clock signal.
Summary of a Simple Troubleshooting Process
Let's try to summarize what we would do to troubleshoot if router 1 were not able to ping router 3. When following this process, you must rectify any unexpected results from the show commands before proceeding to the next stage, such as reconnecting disconnected cables, and correcting incompatible encapsulation types, IP addresses, and routing processes.
  1. Issue the show interface ethernet 0 command on router 1, to make sure that the interface and line protocol are up.
  2. Ping the address of the Ethernet 0 interface on router 2 from router 1.
  3. Telnet into router 2 and issue the show interface serial 0 command and check that the interface and line protocol are up.
  4. Ping router 3 from router 2.
  5. Issue the show ip route command on all three routers to check that a route at the IP level exists between source and destination.
  6. Ping router 3 from router 1.
This process is simplified greatly if you have more than one way to remotely log into a remote router. If your only means to log onto a router located in another city is via the serial interface connection, and that connection is not functioning, troubleshooting options are limited. Having some dial-up facility to a console port, even if it requires local personnel to power on the modem before you can get in, greatly improves your abilities to troubleshoot.
Having taken an overview of troubleshooting, we will examine specific problems and specific troubleshooting commands in more depth. The next section will examine troubleshooting specific interface types. The techniques used to analyze problems in this section will be useful when troubleshooting connectivity for individual segments from source to destination. A subsequent section will examine troubleshooting protocol-specific issues and problems with multilink end-to-end connectivity.
If you can get access to a Cisco router while you read this section, I recommend that you familiarize yourself with the use of the "?" character. While in privileged mode, you can use the "?" character after the show or debug keywords to inform you of what displays are available to give you information on the status of interfaces, protocols, and router processes. An example of using the show ? command is given in Fig. 8-4, and it should be noted that each of the options displayed as a result of the show ? command have sub-options under them. For example, typing show ip ? would display just the show commands for IP.
router3#sho ?
access-expressionList access expression
access-listsList access lists
aliasesDisplay alias commands
arpARP table
asyncInformation on terminal lines used as router interfaces
bridgeBridge Forwarding/Filtering Database [verbose]
buffersBuffer pool statistics
cdpCDP information clockDisplay the system clock
cmnsConnection-Mode networking services (CMNS) information
compressShow compression statistics.
configurationContents of Non-Volatile memory
controllersInterface controller status
debuggingState of each debugging option
dhcpDynamic Host Configuration Protocol status
dialerDialer parameters and statistics
dnsixShows Dnsix/DMDP information
dxiatm-dxi information
entryQueued terminal entries
flashSystem Flash information flh-logFlash Load Helper log buffer
frame-relayFrame-Relay information
historyDisplay the session command history
hostsIP domain-name, lookup style, nameservers, and host table
interfacesInterface status and configuration
ipIP information
lineTTY line information
llc2IBM LLC2 circuit information
loggingShow the contents of logging buffers
memoryMemory statistics
ntpNetwork time protocol
printersShow LPD printer information
privilegeShow current privilege level
processesActive process statistics
protocolsActive network routing protocols
queueShow queue contents
queueingShow queueing configuration
registryFunction registration information
reloadScheduled reload information
rhostsRemote-host+user equivalences
rifRIF cache entries
route-maproute-map information
running-configCurrent operating configuration
sessionsInformation about Telnet connections
sandsSMDS information
snapshotSnapshot parameters and statistics
snapsnap statistics
spanning-treeSpanning tree topology
stacksProcess stack utilization
standbyHot standby protocol information
startup-configContents of startup configuration
subsystemList subsystems
tcpStatus of TCP connections
terminalDisplay terminal configuration parameters
usersDisplay information about terminal lines
versionSystem hardware and software status
whoamiInfo on current tty line
x25X.25 information
Figure 8-4: The show ? screen output to list all show options

 


 
Books24x7.com, Inc © 2000 –  Feedback